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This work was performed by the Jet 
Propulsion Laboratory, California Institute 
of Technology, for the FAA through- -en 
agreement with NASA. (HAS 7-918). 

This presentation is an update 'to.Jthe November 1988 GSFC 
First Ada Symposium presentation which provided preliminary data 
reflecting the RWP Project at the Build-3 Preliminary Design 
Review. This presentation is based upon the completion of the 
RWP Build-3 development and the associated Metrics Report draft. 
The RWP Build-3 Metrics Report will be completed in March 1991 
and will be submitted for public release which may take 3-5 
months. Because this presentation is based on the draft Metrics 
Report, prior to complete validation of all the data,- minor 
corrections may result after the Final Metrics Report is 
completed. 

The development of the RWP System is sponsored by the 
Federal Aviation Administration (FAA) . The RWP is one of several 
weather information programs the FAA has identified in the FAA’s 
National Airspace System (NAS) Plan, which describes all programs 
planned for modernizing and improving air traffic control and 
airway facilities services by the year 2000. 

An integral part of the overall upgraded NAS, the objective 
of the RWP is to improve the quality of weather information and 
the timeliness of its dissemination to system users. To 
accomplish this, an RWP will be installed in each of the Center 
Weather Service Units (CWSUs) , located in 21 of the 23 Air Route 
Traffic Control Centers (ARTCCs) . The RWP System is a Prototype 
System. It is planned that the software will be GFE and that 
production hardware will be acquired via industry competitive 
procurement . 

The ARTCC is a facility established to provide air traffic 
control service to aircraft operating on Instrument Flight Rules 
(IFR) flight plans within controlled airspace, principally during 
the en route phase of flight. Beginning in 1993, and continuing 
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to 1998 , the ARTCCs will be reconfigured to include both en route 
and approach control functions. The reconfigured facilities will 
be called Area Control Facilities (ACFs) . 

rwp will process up to 27 Next Generation Radar (NEXRAD) 
weather data simultaneously in real-time and create mosaic 
displays. The processed NEXRAD data is disseminated directly to 
meteorologists and FAA aircraft controllers. This information is 
updated every three to five minutes. 

The RWP project was started in November of 1987 which 
resulted from the descoping of the Central Weather Processor 
Project (CWP) . At the time of the descoping the CWP was in 
detailed design and planned for the "C" programming language 
development environment. RWP is following DOD-STD-2167A and the 
software will be coded in the DOD standard ADA programming 
language. RWP is composed of 3 incremental development builds 
(Build-1, Build-2 and Build-3) . Build-3 contains all of the 
capabilities specified in the RWP System Specification. There 
was one Preliminary Design Review (PDR) for the entire system and 
an individual Critical Design Review (CDR) for each Build. The 
Coding and Unit Testing (CUT) was completed in February 1990. 
System Testing was completed in June 1990. FAA Prototype (FAA 
Users) Test & Evaluation (PT&E) was completed in July 1990. 
Following PT&E several changes were made to improve the Man- 
Machine Interface and System Reliability. This was followed by 
the FAA Formal System Acceptance Test (FSAT) completed in October 
1990. Final as-built documentation and the FSAT Test Report are 
scheduled for mid January 1991. 

The system is composed of one CSCI developed by JPL that has 
704 Computer Software Units (CSUs) and is composed of 97,687 Ada 
Statements, number of semicolon "? H delimiters, (or 213,961 
Source Lines of Code ( (SL0C) ) , carriage return delimiters less 
comments and blanks, but including specifications and data, type, 
declarations). In addition it has 4,330 of "C" SLOC. 

In addition to the software developed by JPL there are two 
areas where Commercial-Off-The-Shelf (COTS) software is used: 

- Communications Protocols 

- Man-Machine Interface (DECWindows and DECs Forms 
Management System) 

Following are some of the metrics and observations. 
Requirements Metrics and Observations 

The RWP System Specification contains a total of 223 
requirements within 70 pages of the document. On the average 
there are about 3 requirements per page. This does not include 
the specification of the external RWP System-to-System 
interfaces. These are contained in a series of Interface Control 
Documents (ICDs) . The System Specification was approved May 
1988. Any System Specification questions, clarifications, or 
additions were reviewed and negotiated by the RWP System Design 
Teat (SDT) which was composed of key technical lead staff from 
each area (Project Office, System Engineering, Software 
Development, Hardware Development, Test and Operations, Product 
Assurance and Configuration Management) . Results of these 
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meetings vere processed using Project Configuration Management 
procedures and documented in the SDT minutes as "Open Issues." 
During the development 222 Open Issues vere discussed by the SOT 
and approximately 40% vere external system interface issues 
relating to ICDs. The 222 Open Issues resulted in 52 Engineering 
Change Requests (ECRs) and 34 Request for Deviation/Waivers, 

(RDWs) to the System Specification. RDWs vere used as the 
interim method for correcting vording in the RWP System 
Specification. 

Approximately 2/3 of the Open Issues vere generated by the 
Test and Operation Organization (TOO) resulting primarily from 
the preparation of the System Integration and Test Descriptions 
and Procedures. The other 1/3 vere generated by the Softvare 
Development Organization (SDO) . 

The 52 ECRs and 34 RDWs caused a significant revork inpacrt 
late in the development life cycle. 

The conclusion ve have dravn is that if the System 
Integration Test Descriptions and Procedures had been prepared 
earlier in the life cycle, most of the Open Issues vould have 
been initiated and resolved before much of the development vas 
completed or even started and the amount of revork vould have 
been minimized (significantly less) . 

SPFR Metrics and Observations 

1,266 Softvare Problem Failure Reports (SPFRs) vere 
generated vhich vere based upon requirements (Priority 1,2,3); 
see DOD— STD-2 167 A error classification. 

SPFRs reflect all errors reported during softvare (CSCI) cr 
system related requirements resting. The only exception is that 
any errors found during Coding and Unit Testing and CSC 
Integration Testing still outstanding at the start of CSCI 
Requirements Testing vere turned into SPFRs at that time. 

Most notable is the small number of SPFRs (18%) that existed 
at the start of CSCI Requirements Testing and the large % (40%) 
of SPFRs found during System Integration and Testing (SIT) . 
Because of schedule pressures the CSCI Requirements Testing (9% 
of errors) vas deleted for the third incremental Build. This 
explains the small number of errors found (9%) during CSCI 
Requirements Testing and likely contributed to the large number 
found during system level testing (SIT, FSAT-1, FSAT-3 = 51%). 

While there are no specific comparisons or conclusions ve 
are prepared to make on the SPFR code grovth. It may serve as an 
important point of reference to note the code grovth per SPFR for 
embedded systems vhere the memory utilization and margin is 
critical. Our experience over six interim error correction 
Builds is that ve had approximately 8.4 Ada statements of 
increase for each SPFR corrected. This does not provide any 
detail of number of specific amounts of code deleted, changed anri 
added; only the net result. 

During SIT there vere 5 errors reported per 1000 Ada 
statements. A more useful number is the error density per SLOC 
vhich allovs for comparison to numerous density reports on 
previous other developments. It is typical in this phase to see 
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error density rates in the 3 to 10 errors per thousand range with 
the median falling around 5-6 errors per thousand, comparing the 
RWP Project error densities with other Fortran, "C" type 
developments it is our observation that there were fewer 
(approximately 50%) errors during the RWP, SIT that some previous 
projects. Some of this probably is due to the use of Ada. 
However, other factors also contributed such as quality of staff, 
low attrition of staff, etc. 

Based upon the number of work years of effort for CSCI 
Requirements Testing versus the number of work years for SIT, SIT 
was 51% more productive in error generation. This is probably 
exaggerated somewhat due to the deletion of the Build-3 CSCI 
Requirements Testing. 

The metrics of the number of work days to fix an SPFR is 
between 1 . 9 and 2 . 3 work days . The average is 2 . 1 work days per 
SPFR correction. This includes any design, coding, unit testing, 
CSU and CSC integration and delivery of the code to the Project 
Software Library. 

The % of SPFRs fixed that were incorrect or created other 
problems attributable to the fixed code was 3% or less. This 
allowed us to use the 4-6 week period prior to release of Builds 
for various system level tests (SIT, FSAT, PTSE) to continue to 
be used SPFR correction rather verification of the SPFRs fixed. 
With 2-3 months centers for Build deliveries and version updates, 
it provided us with 1/3 more time to fix SPFRs and a higher 
overall SPFR correction productivity rate given a fixed period. 
Ada Portability Wetric3 and Observations 

Ada portability was established as a Project high priority 
design goal. The object was to minimize the various code 
constructs that may need to changed using the same programming 
language and software design but different hardware. The 
following- specific design decisions were made to meet the 
portability goal: 

- Ada Programming Language & Standard 

- Ada Tasking to minimize Operating System function 
uniqueness 

DEC Windows (X-windows subset) to minimize the Man- 
Machine Interface rework 

- Object Oriented Design Methodology to localize external 
interface dependencies and rework 

Other Engineering Principles and Standards to minimize 
rework 

A tool was developed and used by the Product Assurance staff 
to analyze the code to identify each non -port able construct and 
provide summary statistics. Because of the still less than 
stable industry standards on X-windows the tool produced the 
portability results both with DECWindows portable and non- 
portable. 

Portability can be measured any number of ways. 

One of the most useful is to measure the number of specific code 
constructs that run a risk of needing to be modified for 
execution on a different computer than that used for the RWP 
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system (i.e. , DEC) . This measure could then be compared to the 
number of code constructs existing in the developed RWP software. 
The tool does provide the number of non-portable constructs (i.e. 
12,267). However there is no measurement of the number of total 
code constructs in the RWP developed code. There i3 a count of 
the number of Ada statements (i.e. 97,687). There may be 1 or 
more constructs per Ada statement but it is still a useful number 
to quantify the ratio or metric of % portable. If ve divide the 
total Ada statements into the non-portable constructs we get the 
answer approx. 12. 5%. Therefore, on a construct basis, the RWP 
system is at least 87.5% portable. This does not include any 
changes needed to accommodate word size or reformatting to 
accommodate storage devices that are unique. It should be 
cautioned that using the difficulty classification to compute 
work hours to port the system should not be done. Since many of 
the porting changes for one type of construct is mechanically 
repeatable and represents a single instance, worst case, the 
estimation of porting effort needs to consider repeatability. In 
addition, not all of the constructs identified as a porting risk 
may need to be ported. 

However, the metrics and analysis should set an industry 
reference point for specifying design requirements for 
portability. 
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JPL RUP METRICS AND OBSERVATIONS RWP 

o DATA IS PRELIMINARY N 'v ! 

o MINOR CORRECTIONS MAY RESULT AFTER VALIDATION PROCESS 

o METRICS REPORT TO BE COMPLETE IN MARCH 1991 

- MILL START PROCEDURE FOR PUBLIC RELEASE OF METRICS REPORT 

- RELEASE MAY TAKE 3 - 5 HONTHS 
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RWP METRICS AND OBSERVATIONS 

JPL 

AGENDA 

o WHAT IS THE RWP SYSTEM? 
o REQUIREMENTS ISSUES 
o TESTING EFFECTIVENESS 
o ERROR DENSITY AND DISCOVERY RATE 
o ADA ERROR CORRECTION RATES 
o PORTABILITY ISSUES 
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RWP METRICS AND OBSERVATIONS 


WHAT IS THE RWP SYSTEM? 

o SPONSOR: FEDERAL AVIATION ADMINISTRATION (FAA) 

o PROTOTYPE DEVELOPMENT; EVENTUALLY PART OF NATIONAL AIRSPACE SYSTEM 
UPGRADE 

o RWP MILL PROCESS WEATHER DATA IN REAL-TIME BY CREATING A MOSAIC DISPLAY 
OF UP TO 27 RADARS SIMULTANEOUSLY. THE DATA WHICH IS DISSEMINATED 
DIRECTLY TO THE FAA AIRCRAFT CONTROLLERS AND METEOROLOGISTS IS UPDATED 
EVERY THREE TO FIVE MINUTES 

o PROJECT MILESTONES: 

- PROJECT START - NOVEMBER 1987 

- CODING COMPLETE - FEBRUARY 1990 

- SYSTEM TESTING COMPLETE •• JUNE 1990 

- FAA OPERATIONAL TEST AND EVALUATION - JULY 1990 

- FAA FORMAL SYSTEM, ACCEPTANCE TEST - OCTOBER 1990 

o 1 RWP SYSTEM AT 21 OF 23 AREA CONTROL FACILITIES; 7 EXTERNAL INTERFACES 
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RWP METRICS AND OBSERVATIONS 

JPL 

WHAT IS THE RUP SYSTEM? (Cont'd) 

S/M INTENSIVE; H/U OFF-THE-SHELF 

- 1 COMPUTER S/U CONFIGURATION ITEH 

• DEVELOPED BY JPL: 97,687 (ADA STATEMENTS) 

213,961 (CARRIAGE RETURNS (COMMENTS 
AND BLANKS)) 

4,330 (C SLOC) 

•COMMERCIAL OFF-THE-SHELF: 280,238 (C SLOC) 

— COMMUNICATIONS PROTOCOLS 
- DEC-UINDOUS 

— DEC FORMS MANAGEMENT SYSTEH 



ADA, DOD-STD-2167, REVISION A: TAILORED 
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WHAT IS THE RWP SYSTEM? (Cont'd) 

- DISTRIBUTED H/U ARCHITECTURE 

• 10 MICRO VAX IIS, 3 MICRO VAX 3600S, 1 MICRO VAX 3200 

• VAXELH AND VAX/VMS OPERATING SYSTEMS, DECNET, ISO PROTOCOLS 

- THE RUP SYSTEMS ARE SCHEDULED TO BE INSTALLED IN THE FAA CONTROL 
CENTERS BY 1994 BY A FAA SYSTEM CONTRACTOR WHO IS SCHEDULED FOR 
SELECTION IN 1992. JPL IS PLANNING TO PROVIDE TECHNICAL SUPPORT TO 
THE FAA THROUGH 1994 FOR THE INSTALLATION OF THE FIRST THREE OF 

23 SITES. 
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RWP METRICS AND OBSERVATIONS 

REQUIREMENTS ISSUES 

0 SYSTEM SPECIFICATION (WRITTEN BY JPL AND FAA) 

- 205 FUNCTION AND PERFORMANCE REQUIREMENTS 

- + 18 PERFORMANCE (COUNTED AS 1) 

*223 

o SYSTEM SPECIFICATION FUNCTIONAL AND PERFORMANCE REQUIREMENTS PAGES = 70 
APPROXIMATELY 0.3 PAGES/REQUIREMENT 

o PROJECT SYSTEM DESIGN TEAM (SDT) ADDRESSED REQUIREMENTS ISSUES AT WEEKLY 
HEETINGS 



0 ISSUE RESOLUTIONS WERE DOCUMENTED IN DESIGN TEAM MINUTES AND PROCESSED 
VIA CONFIGURATION MANAGEMENT: 

- ENGINEERING CHANGE REQUESTS (ECRs) TO SRS, ICDs AND SYSTEH/SEGMEHT 

If DESIGN DOCUMENT 

- REQUEST FOR DEVIATION/WAIVER (ROW) TO SYSTEH SPECIFICATION 
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RWP METRICS AND OBSERVATIONS 

JPL 

REQUIREMENTS ISSUES (Cont'd) 

222 OPEN ISSUES DISCUSSED AT DESIGN TEAM 

- APPROXIMATELY 40* WERE INTERFACE (ICO) ISSUES 

RESOLUTION RESULTED IN: 

- 52 ECRs TO ICDs, SRS AND SSDD 

- 34 ROMs TO SYSTEM SPECIFICATION 

APPROXIMATELY TUO-THIRDS OF OPEN ISSUES CAME FROM SIT STAFF DOING STT 
DESCRIPTIONS AND PROCEDURES (SITD/P) 

MOST ECRs AND RDUs RESULTED IN SOFTWARE, DOCUMENT AND TEST PROCEDURE AND 
DATA REWORK 

WRITE INITIAL VERSION OF SYSTEM TEST PR OCEDURES 
AS_EARLY_AS_EOSSIBLE 
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RUP METRICS AND OBSERVATIONS 

JPL 

SPFR COUNT 

(by phase) 

o TOTAL APPROXIMATELY 2, 100 SPFRs TO DATE 

1,452 RUP CSCI RELATED 
1,266 PRIORITY 1 - 3 


PHASE 

SPFR COUNT 

PERCENT 

SYSTEM INTEGRATION TESTING 

508 

401 

CSCI REQUIREMENTS TESTING 

117 

9* 

CSC INTEGRATION TESTING 

231 

181 

CODE AND UNIT 

6 

0* 

FSAT-1 

98 

8V 

FSAT-2 

35 

31 

BUILD 

172 

14% 

OTHER 

98 

8% 


17255 

irat 
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RWP METRICS AND OBSERVATIONS 


RUP SPFR DISTRIBUTION (BY PHASE) 


cur BUILD I 


cur BUILD 2 


CUT BUILD 3 


CSC! REQ TEST 


5Y5TEI1 TEST 


CSC DD I 


CSC QD 2 


CSC BD 3 
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RUP 

METRICS AND OBSERVATIONS 
SPFR DENSITY 

BWP 

Mr/' 

BUILD 

CARRIAGE 

RETURNS 

SEMI- 

COLONS 

.FROM PREVIOUS BUIU> » FROM PREVIOUS E 
(CARRIAGE RETURNS) (SEMI-COLONS. 

IUILD SPFRS 
) FIXED 

3.8 

401,544 

94,886 

mm m w 

— 

128 

3.9 

405,537 

95,613 

3,993 

727 

55 

3.10 

408,895 

96,214 

3,358 

601 

80 

3.11 

409,362 

96,308 

467 

89 

18 

3.12 

414,546 

97,687 

5,164 

1,379 

78 

3.13 

420,804 

99,288 

6,258 

1,601 

135 

3.14 

418,433 

98,748 

-2,371* 

-504* 

95 

TOTAL 6 BUILDS 

(3.9 - 3 

.14) 16,889 

3,857 

461 




SEMI-COLONS PER SPFR 

8.4 





CARRIAGE RETURNS PER SPFR 

36.6 
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RWP METRICS AND OBSERVATIONS 

ERROR DENSITY AND DISCOVERY RATES 

o CSCI REQUIREMENTS TESTING 

- STOPPED FOR BUILD-3: SCHEDULE AND RETURN ON INVESTMENT 

- MOVED TO SDO FOR BUILD-4 

• PRODUCTIVITY: LESS OVERHEAD TO ERROR PROCESSING 

• EMPHASIZE REQUIREMENTS RESPONSIBILITY OF SDO STAFF 





o SYSTEM INTEGRATION AND TESTING (SIT) 

- NINE MONTHS TEST EXECUTION PERIOD 

- RESET AFTER THREE MONTHS TO ACCOMMODATE LATE SOFTWARE DELIVERY 

- 401 OF ERRORS FOUND DURING SIT 
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RWP METRICS AND OBSERVATIONS 
jpl RWP 

ERROR DENSITY AND DISCOVERY RATES (Cont'd) ''W 

o FORMAL ACCEPTANCE SYSTEH (FSAT) 

- FAA TEST WITNESS 

- lit OF SPFRs FOUND DURING FSAT 

- TWO FSATs 

• FSAT-1: APPROXIMATELY THREE WEEKS: 98 SPFRs <8% SPFRs) 

• FSAT-2: APPROXIMATELY ONE WEEK: 35 SPFRs (3t SPFRs) 


o 95+t SYSTEM FUNCTION AND PERFORMANCE REQUIREMENTS FULLY VALIOATEO 


o SIT HETRICS AND OBSERVATIONS 

- APPROXIMATELY 5 ERRORS PER 1000 ADA STATEMENTS 

- APPROXIMATELY 2.3 ERRORS PER 1000 CARRIAGE RETURNS 


APPROXIMATELY 1/2 LESS THAN TYPICAL FORTRAN 
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RWP METRICS AND OBSERVATIONS 




ERROR DENSITY AND DISCOVERY RATES (Cont'd) 



o SIT VERSUS CSCI REQUIREMENT TESTING (NOTE-1) 

- CSCI REQUIREMENTS TESTING APPROXIMATELY 19. 5 ERRORS/TEST WORK YEAR 

- SYSTEM INTEGRATION AND TESTING APPROXIMATELY 29.5 ERRORS/TEST WORK 
YEAR 


SIT APPROXIMATELY 511 MORE PRODUCTIVE THAN CSCI 
REQUIREMENTS TESTING 


NOTE-1 : SYSTEM = 1 CSCI 
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RWP METRICS AND OBSERVATIONS 

ADA ERROR CORRECTION RATES 


NUMBER OF SPFRs 
IN BUILD 

WORK DAYS 
_PER/SPFR 


108 

1.5 


128 

2.2 

2-3 MEEKS/BUILD 

55 

2.1 


80 

1.9 

8 BUILDS OVER 6 MONTHS 

98 

2.6 


78 

2.3 

EXPERIENCED RUP/ADA STAFF 

135 

2.3 


95 

1.9 



16.9 = 2.1 AVERAGE WORK DAYS/SPFR CORRECTION 

o TYPICAL WORK DAYS PER SPFR APPROXIMATELY 1.9 TO 2.3 
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RWP METRICS AND OBSERVATIONS 

JPL 

ADA ERROR CORRECTION RATES (Cont'd) 
o 96+% OF CORRECTIONS HERE VALID: 

- UE WERE ABLE TO HAKE ONE ADDITIONAL BUILD PRIOR TO FSATs TO 

INCREASE RELIABILITY 
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ADA PORTABILITY METRICS AND OBSERVATIONS 
0 PROJECT ESTABLISHED PORTABILITY AS DESIGN OBJECTIVE EARLY 
o PERFORMED ANALYSIS USING THREE TOOLS AND LIMITED HUMAN ANALYSIS 

- ADA COMPILER 

- JPL DEVELOPED TOOL: 

* SEE PAPER BY BORIS SHENKER AND HERHAN GUARDA 

AN AUTOMATED TOOL FOR PORTABILITY ANALYSIS OF ADA CODE OF THE 
REAL-TIME WEATHER PROCESSOR PROJECT 
PRESENTED AT NINNOWBROQK WORKSHOP, JULY 1990 

- ADA-MAT: FOR VALIDATION 



0 
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PORTABILITY HAS THREE LEVELS OF RISK: 

EFFORT TO CONVERT 
-LOW 0-2 WORK HOURS 

- MEDIUM 2-8 WORK HOURS 

- HIGH OVER 8 WORK HOURS 
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ADA PORTABILITY METRICS 
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TOTAL UNITS 

PORTABLE UNITS 

NON-PORTABLE UNITS 

UNITS WITH HIGH RISK CONSTRUCTS 

UNITS ONLY WITH LOVJ RISK CONSTRUCTS 

TOTAL ADA STATEMENTS (;) 

TOTAL NON-PORTABLE CONSTRUCTS: 

- HARDWARE 

- OPERATING SYSTEM** 

- ADA COMPILER 

- COMMERCIAL OFF-THE-SHELF (COTS) 


100 % 


97,687 

12,267 

1,192 

2,290 

5,220 

3,565 


100 % 


DOES NOT INCLUDE DATA ISSUES (E.G. WORD SIZE, STORAGE ISSUES) 


DOCS NOT INCLUDE PARAMETER SETTINGS 
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ADA PORTABILITY METRICS 

X-MIHDOWS PORTABLE 
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* DIFFERENCE 


TOTAL NON-PORTABLE CONSTRUCTS 

11,444 

n 

HARDWARE 

1,192 

0 

OPERATING SYSTEM** 

2,290 

0 

ADA COMPILER 

5,220 

0 

COTS 

2,742 

5 * 
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